Skip to content

feat(hgraph): restore conjugate graph enhancement - #2675

Open
wxyucs wants to merge 1 commit into
antgroup:mainfrom
wxyucs:codex/issue-2650-hgraph-enhancement
Open

feat(hgraph): restore conjugate graph enhancement#2675
wxyucs wants to merge 1 commit into
antgroup:mainfrom
wxyucs:codex/issue-2650-hgraph-enhancement

Conversation

@wxyucs

@wxyucs wxyucs commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add HGraph-scoped conjugate-graph build/search parameters and maintained Feedback/Pretrain support
  • integrate label updates, result enhancement, memory accounting, and binary/streaming serialization
  • make exact feedback scans safe during concurrent Add by atomically publishing initialized I/O extents
  • preserve older binary and streaming indexes when loading into a conjugate-enabled target, while rejecting data loss in the reverse direction
  • bound exact scans to published logical code counts for deduplicated and regular storage
  • migrate the runnable example and synchronize active English/Chinese HGraph documentation

Validation

  • clang-format 15
  • clang-tidy 15 on changed implementation files
  • focused HGraph parameter, ConjugateGraph, I/O, unsupported-default, and enhancement functional tests
  • focused TSAN concurrent Add/Feedback regression
  • HGraph enhancement example (recall 0.87 to 1.00)

Fixes: #2650

Copilot AI lite review requested due to automatic review settings August 12, 2026 10:14
@vsag-bot

vsag-bot commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @inabao

@mergify

mergify Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

  • label~=^kind/

🟢 Require version label

  • label~=^version/

🟢 Require linked issue for feature/bug PRs

  • body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

@wxyucs wxyucs added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/1.1 labels Aug 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Restores and integrates the HGraph conjugate-graph enhancement end-to-end (parameters, feedback/pretrain APIs, search-time result enhancement, and binary/streaming serialization), plus updates example and EN/ZH docs.

Changes:

  • Added HGraph build/search parameters to enable conjugate-graph construction and toggled search-time usage.
  • Implemented Feedback/Pretrain/UpdateId conjugate-graph integration, including streaming/binary serialization and memory accounting.
  • Migrated the C++ runnable example and synchronized English/Chinese documentation for the feature.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/test_hgraph.cpp Adds functional coverage for conjugate-graph feedback, ID updates, and serialization roundtrips.
src/storage/serialization_tags.h Introduces a new streaming block tag for conjugate-graph payloads.
src/impl/conjugate_graph.h Adds StreamWriter-based serialization API for ConjugateGraph.
src/impl/conjugate_graph.cpp Implements StreamWriter serialization for conjugate-graph state and footer.
src/algorithm/hgraph/hgraph_serialize.cpp Serializes/deserializes conjugate-graph in binary + streaming formats and exposes metadata/memory usage.
src/algorithm/hgraph/hgraph_search.cpp Enhances search results using conjugate-graph edges when enabled.
src/algorithm/hgraph/hgraph_parameter_test.cpp Adds unit tests for parameter mapping and search parameter parsing.
src/algorithm/hgraph/hgraph_parameter.h Adds new build/search parameter fields for conjugate-graph enablement and search toggle.
src/algorithm/hgraph/hgraph_parameter.cpp Parses/serializes the new parameters and includes them in compatibility checks.
src/algorithm/hgraph/hgraph_param_mapping.cpp Maps external param key(s) for conjugate-graph into internal HGraph config.
src/algorithm/hgraph/hgraph_enhance.cpp Implements Feedback, Pretrain, and UpdateId conjugate-graph wiring.
src/algorithm/hgraph/hgraph.h Adds conjugate-graph fields/mutex and new public API overrides.
src/algorithm/hgraph/hgraph.cpp Initializes conjugate-graph based on build params.
src/algorithm/hgraph/CMakeLists.txt Adds the new enhancement implementation unit to the build.
examples/cpp/304_feature_enhance_graph.cpp Updates the example to HGraph + new parameter structure.
docs/docs/zh/src/resources/index_parameters.md Documents use_conjugate_graph and use_conjugate_graph_search (ZH).
docs/docs/zh/src/indexes/hgraph.md Documents the new build parameter (ZH).
docs/docs/zh/src/advanced/enhance_graph.md Updates enhancement docs to HGraph APIs and new JSON structure (ZH).
docs/docs/en/src/resources/index_parameters.md Documents use_conjugate_graph and use_conjugate_graph_search (EN).
docs/docs/en/src/indexes/hgraph.md Documents the new build parameter (EN).
docs/docs/en/src/advanced/enhance_graph.md Updates enhancement docs to HGraph APIs and new JSON structure (EN).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp Outdated
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_serialize.cpp
Comment thread examples/cpp/304_feature_enhance_graph.cpp Outdated

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review for feat(hgraph): restore conjugate graph enhancement

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Copilot AI review requested due to automatic review settings August 12, 2026 11:03
@wxyucs
wxyucs force-pushed the codex/issue-2650-hgraph-enhancement branch from 09c0a4d to 98d83af Compare August 12, 2026 11:03

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by vsag-pr-review-agent

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review for PR #2675.

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/algorithm/hgraph/hgraph_search.cpp:754

  • In the conjugate-graph enhancement path, results are converted inner_id -> label and then label -> inner_id again. When duplicate external IDs are enabled, LabelTable can have multiple inner IDs for the same label, and TryGetIdByLabel() returns an arbitrary one. That can cause mismatched (distance, inner_id) pairs (distances from one duplicate but extra_info/inner_id from another) when pushing back into search_result.
        while (not label_results.empty()) {
            const auto record = label_results.top();
            label_results.pop();
            const auto [found, inner_id] = this->label_table_->TryGetIdByLabel(record.second, true);
            if (found and (ft == nullptr or ft->CheckValid(inner_id))) {

examples/cpp/304_feature_enhance_graph.cpp:67

  • If CreateIndex() fails, hgraph remains null but is still dereferenced (hgraph->Build), which will crash the example. The example should exit/return on create failure and ideally surface the factory error message.
    std::shared_ptr<vsag::Index> hgraph;
    if (auto index = vsag::Factory::CreateIndex("hgraph", hgraph_build_parameters);
        index.has_value()) {
        hgraph = index.value();
    } else {

@wxyucs
wxyucs force-pushed the codex/issue-2650-hgraph-enhancement branch from 98d83af to 4844741 Compare August 12, 2026 11:27
Copilot AI review requested due to automatic review settings August 12, 2026 11:27

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR restores conjugate graph enhancement for HGraph with comprehensive coverage: Feedback/Pretrain methods, search-time enhancement integration, UpdateId support, memory accounting, and both binary and streaming serialization. The implementation is well-structured and the test coverage is thorough.

Issues from previous review rounds

All previously reported issues have been addressed:

  • Lock ordering: label_lookup_mutex_ is consistently acquired before conjugate_graph_mutex_ across all code paths (Feedback, Pretrain, UpdateId, search enhancement).
  • Filter semantics: The enhancement result rebuild now holds a shared label lock and re-checks ft->CheckValid(inner_id) before pushing results back.
  • Performance: flatten and computer are created once outside the per-label lambda.
  • Streaming deserialization: Now correctly rejects a conjugate-graph block when the target HGraph was created without the feature, matching binary deserialization behavior.
  • Typo fix: "constains" → "contains" in the example.

Current assessment

The code is clean, well-tested, and all critical concerns from earlier rounds have been resolved. No new issues found in this revision.

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/impl/conjugate_graph.cpp

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Suppressed comments (4)

src/impl/conjugate_graph.cpp:185

  • The streaming format writes adjacency entries without an explicit conjugate_graph_ entry count, which makes the payload harder to evolve and validate (the reader must infer termination via footer mechanics). Consider serializing the number of nodes/entries up front (e.g., conjugate_graph_.size()) so deserialization can be bounded and format changes are easier to manage.
    StreamWriter::WriteObj(out_stream, memory_usage_);
    for (const auto& [tag_id, neighbor_ptr] : conjugate_graph_) {
        StreamWriter::WriteObj(out_stream, tag_id);
        uint64_t neighbor_set_size = neighbor_ptr->size();
        StreamWriter::WriteObj(out_stream, neighbor_set_size);
        for (const auto neighbor_tag_id : *neighbor_ptr) {
            StreamWriter::WriteObj(out_stream, neighbor_tag_id);
        }
    }

src/impl/conjugate_graph.cpp:190

  • Serializing the footer through a temporary std::stringstream forces an extra buffering + string copy (str()), which can be noticeable for large serialized payloads or frequent serialization. Consider adding a Footer::Serialize(StreamWriter&) (or writing the footer into a pre-sized buffer once) to avoid the extra allocation/copy.
    std::stringstream footer_stream(std::ios::in | std::ios::out | std::ios::binary);
    footer_.Serialize(footer_stream);
    const auto footer_data = footer_stream.str();
    out_stream.Write(footer_data.data(), footer_data.size());
}

src/algorithm/hgraph/hgraph_search.cpp:754

  • The label_lookup_mutex_ shared-lock is held across potentially expensive work (flatten->Query(...) and EnhanceResult(...)). This can increase lock contention and delay concurrent operations that need the label table (e.g., updates). Consider narrowing the lock scope: extract/copy the needed label mappings (or the initial label list) under the lock, then release it before computing distances and enhancing results; reacquire only for the final TryGetIdByLabel re-mapping (or precompute those mappings too).
        std::shared_lock label_lock(this->label_lookup_mutex_);
        while (not search_result->Empty()) {
            const auto record = search_result->Top();
            search_result->Pop();
            label_results.emplace(record.first, this->label_table_->GetLabelById(record.second));
        }

        const auto flatten = use_custom_distance ? nullptr : this->get_precise_codes();
        const auto computer = use_custom_distance ? nullptr : flatten->FactoryComputer(raw_query);
        const auto distance_of_label = [&](int64_t label) {
            const auto [found, inner_id] = this->label_table_->TryGetIdByLabel(label, true);
            if (not found or (ft != nullptr and not ft->CheckValid(inner_id))) {
                return std::numeric_limits<float>::max();
            }
            float distance = std::numeric_limits<float>::max();
            if (use_custom_distance) {
                request.distance_batch_func_(&label, 1, &distance);
            } else {
                flatten->Query(&distance, computer, &inner_id, 1, &ctx);
            }
            if (request.threshold_.has_value() and distance > request.threshold_.value()) {
                return std::numeric_limits<float>::max();
            }
            return distance;
        };
        {
            std::shared_lock graph_lock(this->conjugate_graph_mutex_);
            (void)this->conjugate_graph_->EnhanceResult(label_results, distance_of_label);
        }
        while (not label_results.empty()) {
            const auto record = label_results.top();
            label_results.pop();
            const auto [found, inner_id] = this->label_table_->TryGetIdByLabel(record.second, true);
            if (found and (ft == nullptr or ft->CheckValid(inner_id))) {
                search_result->Push(record.first, inner_id);
            }
        }

examples/cpp/304_feature_enhance_graph.cpp:69

  • If CreateIndex(...) fails, the example continues and will dereference hgraph later (e.g., hgraph->Build(...)), which will crash. The example should exit/return on failure (or throw) right after printing the error.
    std::shared_ptr<vsag::Index> hgraph;
    if (auto index = vsag::Factory::CreateIndex("hgraph", hgraph_build_parameters);
        index.has_value()) {
        hgraph = index.value();
    } else {
        std::cout << "Create HGraph Error" << std::endl;
    }

Comment thread src/algorithm/hgraph/hgraph_serialize.cpp Outdated
Comment thread examples/cpp/304_feature_enhance_graph.cpp Outdated
Comment thread src/impl/conjugate_graph.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

经过对最新 commit (3311525) 的全面审查,确认之前 Copilot 和其他 reviewer 发现的所有重要问题都已在修正后的代码中得到解决:

已确认修复的问题:

  1. 锁顺序问题 — Feedback 中现在正确使用 label_lookup_mutex_ (shared) → conjugate_graph_mutex_ (unique) → memory_usage_mutex_ (unique) 的顺序
  2. 过滤语义 — conjugate graph 增强后重新检查 is_allowed(inner_id) 再 push 回结果
  3. flatten/computer 性能 — 在 lambda 外部创建,避免重复构造
  4. label-table 无锁访问 — Pretrain 中已使用 TryGetIdByLabel 并持有 label_lookup_mutex_
  5. streaming deserialization 一致性 — 现在 streaming 路径也拒绝 mismatched conjugate graph 块
  6. 示例 typo — 已修正
  7. memory_usage_ 溢出 — AddNeighbor 中已添加 overflow check
  8. UpdateId 锁范围 — 已缩小为仅包裹 conjugate graph 更新

代码质量评估:

  • 新文件 hgraph_enhance.cpp 结构清晰,错误处理完善
  • 序列化支持完整(binary + streaming),版本兼容性处理得当
  • 测试覆盖全面:基本功能、并发安全、range search 不受影响、参数解析、序列化往返
  • 内存统计正确集成到 cal_memory_usage()GetMemoryUsageDetail()
  • 参数映射遵循现有模式,CheckCompatibility 正确包含 use_conjugate_graph

没有发现新的阻塞性问题。

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread examples/cpp/304_feature_enhance_graph.cpp Outdated
Copilot AI review requested due to automatic review settings August 24, 2026 06:08
@wxyucs
wxyucs force-pushed the codex/issue-2650-hgraph-enhancement branch from 3311525 to f27fbe9 Compare August 24, 2026 06:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Copilot AI review requested due to automatic review settings August 24, 2026 09:02
@wxyucs
wxyucs force-pushed the codex/issue-2650-hgraph-enhancement branch from f27fbe9 to a696d62 Compare August 24, 2026 09:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
@wxyucs

wxyucs commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

The failed Test Example X86 job is unrelated to this PR and is reproducibly flaky:

  • examples/cpp/327_feature_autotune_existing_index.cpp is byte-for-byte identical to main (it was introduced by feat: add AutoTune V1 orchestration #2569); this PR does not modify it.
  • The failed job log shows the example created HGraph with "use_conjugate_graph": false, so the new enhancement/search path was disabled.
  • Repeating the exact example binary built from PR SHA a696d627 failed 14/100 runs: recall was 0.75 in 10 runs, 0.5 in 3, and 0.0 in 1; the other 86 runs reached 1.0. The Actions failure (0.75) is the same stochastic low-recall mode.
  • All other jobs in the run passed, including X86 unit/functional tests, TSAN, compatibility, formatting, macOS arm64, CircleCI, DCO, and merge protections.

No PR code change is warranted. I am rerunning only failed job 97400471144.

@wxyucs

wxyucs commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Control confirmation: I also built and ran the example 100 times from the PR base / current main SHA c7b6a54e, with none of this PR’s changes present. It likewise failed 14/100 runs (8 at recall 0.75 and 6 at 0.5); the other 86 reached 1.0. This independently confirms the failure is pre-existing and unrelated to #2675.

Signed-off-by: Xiangyu Wang <wxy407827@antgroup.com>
Assisted-by: Codex:gpt-5
Copilot AI review requested due to automatic review settings August 24, 2026 13:21
@wxyucs
wxyucs force-pushed the codex/issue-2650-hgraph-enhancement branch from a696d62 to ac5023e Compare August 24, 2026 13:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_search.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Comment thread src/algorithm/hgraph/hgraph_enhance.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 module/api module/docs module/example module/testing size/XL version/1.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat](hgraph): restore conjugate-graph enhancement and Pretrain support

4 participants